NCEE EVALUATION BRIEF 


September 2013 





NATIONAL CENTER for 

EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 


Institute of Education Sciences 


AFTER TWO YEARS, THREE ELEMENTARY 
MATH CURRICULA OUTPERFORM A 
FOURTH 


Which math 
curriculum 
should I use? 


This brief aims to help educators understand the implications of math curriculum choice in the 
early elementary grades by presenting new findings from a study that examined how four math 
curricula affect students’ achievement across two years— from 1st through 2nd grades. 1 The four 
curricula were (1) Investigations in Number, Data, and Space (Investigations); (2) Math 
Expressions; (3) Saxon Math (Saxon); and (4) Scott Foresman-Addison Wesley Mathematics 
(SFAW), which the developer revised and renamed enVision Math (enVision) during the study. 
These curricula are widely used and differ in their approaches to teaching and learning. Within 
districts, we randomly assigned one of the four curricula to each school that participated in the 
study. After one year (by the end of 1st grade), students taught with Math Expressions and Saxon 
made greater gains in achievement than students taught with Investigations and SFAW. After two 
years (by the end of 2nd grade), Investigations students continued to lag behind Math 
Expressions and Saxon students, while SFAW/enVision students caught up to Math Expressions 
and Saxon students. Therefore, Math Expressions, Saxon, and SFAW/enVision improved lst- 
through-2nd- grade math achievement by similar amounts, and all three outperformed 
Investigations. Our findings also suggest that switching between some of the study ’s curricula 
does not harm student achievement and can even be beneficial. 
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AFTER TWO YEARS, THREE ELEMENTARY MATH CURRICULA OUTPERFORM A FOURTH 


The selection of a math curriculum affects the nature of mathematics instruction and, 
ultimately, the opportunities for students to learn mathematics. According to the National 
Research Council (2001), “Opportunity to leam is widely considered the single most important 
predictor of student achievement.” Research confirms this important connection between 
curricula and achievement. For example, at the elementary level, previous findings issued by this 
study indicate that some curricula improve the average student’s percentile rank in math by as 
much as 12 points more than other curricula during the first year of implementation (Agodini et 
al. 2009, 2010). 

The choice of curriculum is particularly critical for elementary schools because they tend to 
use one math curriculum in several grades, thus affecting students for multiple years. However, 
little is known about how these curricula affect students over time. This is a key issue because 
curricula differ in important ways that could have implications for students’ learning over time 
and, therefore, their effects may differ from those found after a single year. For example, 
differences between curricula in the sequencing of concepts, the extent of review of previously 
learned skills, or the alignment with student assessments in different grades could lead to 
achievement effects that vary from year to year. Also, some curricula place higher cognitive 
demands on students, by having them figure out important mathematical ideas that typically are 
not immediately apparent. This practice helps students develop conceptual understanding of 
math that could improve gradually over several years (Hiebert and Grouws 2007). Finally, some 
curricula are clearer about the objectives of the lessons and provide better teacher supports for 
implementing the lessons (Stein and Kaufman 2010), which could in turn lead to differing effects 
across years and grades. 

To help educators more fully understand this issue, we expand on previous findings from 
this study by examining how four math curricula affect students’ achievement across two 
years — from 1st through 2nd grades: (1) Investigations in Number, Data, and Space 
(Investigations); (2) Math Expressions; (3) Saxon Math (Saxon); and (4) Scott Foresman- 
Addison Wesley Mathematics (SFAW), which the developer revised and renamed enVision 
Math (enVision) during the study.” These curricula are widely used in the early elementary 
grades 3 and, as described below, differ in their approaches to mathematics instruction. 

When making curricular decisions, it is important to consider not only the potential effects 
of the new curriculum, but also the implications of switching from one curriculum to another. 
Because many districts make their own curricular decisions, and some districts let schools 
choose their own curriculum, students who move from one district or school to another could 
switch curricula. This issue is of particular concern for disadvantaged students, who are more 
likely to transfer between schools (Rumberger 2003). Switching curricula also could be an issue 
for students who do not transfer between schools because many districts review their elementary 
math curriculum every several years and often adopt a new one at that time, resulting in a large 
fraction of students experiencing more than one math curriculum during their elementary years. 4 
For example, California, Florida, and Texas are among 22 states with a textbook adoption 
process that every six years or so determines a selection of math texts that districts can purchase 
with state funds, and districts in these states tend to follow this curriculum adoption schedule. 5 
Our study also provides evidence that helps educators understand the effects on student 
achievement of switching from one curriculum to another. 
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This brief addresses three questions: 

1. Do some of the study’s curricula improve lst-through-2nd-grade student math 
achievement more than others? 

2. How do these curriculum-group differences in achievement after two years (by the 
end of 2nd grade) compare with the curriculum-group differences after one year (by 
the end of 1st grade)? 

3. How does switching curricula (instead of staying with the same one) affect student 
achievement during the first year a new curriculum is used? 

In the remainder of this brief, we provide an overview of the study design, including the 
selection, description, and assignment of study curricula. Next, we examine curriculum 
implementation to place the student achievement effects in context. We then present our findings 
on the achievement effects of elementary math curricula after one and two years, along with the 
effects of switching curricula. Finally, we conclude with a discussion and suggestions for future 
research. 

Study Design 

To set up the study, we (1) selected the curricula to be included, (2) documented the key 
ways in which the programs differ, and (3) randomly assigned the curricula to schools that 
agreed to participate in the study. 

Selecting the Curricula 

We used a competitive process to choose our curricula. Developers and publishers of math 
curricula for early elementary school were invited to submit proposals to have their curricula 
included in the evaluation. The goal of this process was to select curricula that represented varied 
approaches to instruction and learning and were likely to improve student achievement more 
than other curricula. To meet this objective, we used six criteria to review the submissions: (1) 
research support for the curriculum’s conceptual framework, (2) empirical evidence of 
effectiveness, (3) teacher practices and skills that comprise the curriculum, (4) quality of training 
and materials, (5) institutional capacity to train the number of teachers in the study, and (6) 
appropriateness of the curriculum for early elementary students in Title I schools. An 
independent panel of experts in math and math instruction reviewed the submissions and 
recommended curricula they believed were suitable for the study. In-person meetings were held 
with publishers whose curricula were considered strong candidates, after which IES selected the 
four curricula mentioned above. 6 

Important Differences Between the Curricula 

To summarize the differences between the curricula, we conducted a comparative analysis 
that was framed using research on effective mathematics instruction. This research has found that 
three aspects of curricula are likely to have a strong influence on instruction (that could affect 
student achievement): (1) mathematical emphasis, (2) instructional approach, and (3) supports 
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for teachers. For each aspect, our review of curriculum materials revealed substantial variation 
across the four curricula, as expected. Below is a summary of the differences, and more detail is 
provided in the appendix. 
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Mathematical emphasis concerns the kinds of mathematics that students had opportunities 
to learn and how those opportunities were structured. To assess this feature, including the ways 
in which the curricula address conceptual understanding and procedural fluency, we examined 

Q 

three components: (1) the cognitive demand of the tasks ; (2) regular routines that provide 
opportunities for engagement with concepts, facts, and procedures; and (3) repeated practice of 
skills and procedures to develop procedural fluency. 9 Cognitive demand refers to the extent to 
which the mathematics tasks involved connections between concepts, procedures, and facts and 
provided potential opportunities to struggle with mathematics. 10 High-demand tasks are 
intellectually and conceptually challenging, whereas low-demand tasks focus on routine and 
procedural elements without connections to concepts. 

As Table 1 shows, nearly all (95 percent) of Investigations’ and Math Expressions’ tasks are 
high demand, compared with 65 percent for Saxon and 57 percent for SFAW/enVision. Looking 
at daily routines and use of repeated practice, we found that Saxon and, to a lesser extent, Math 
Expressions both emphasize procedural fluency more than Investigations and SFAW/enVision 


do. 

Table 1. Summary of Key Curriculum Differences 



investigations 

Math Expressions 

Saxon 

SFAW/enVision 

Mathematical Emphasis 

Cognitive Demand of the 
Primary Tasks in Each 
Curriculum 

DM - 40% 
PWC - 55% 
PWOC - 5% 
M - 0% 

DM - 30% 
PWC - 65% 
PWOC - 0% 
M - 5% 

DM - 0% 
PWC - 65% 
PWOC - 35% 
M - 0% 

DM - 0% 
PWC - 57% 
PWOC - 40% 
M - 3% 

Frequency and Length of 
the Routine 

Daily 

10 minutes 

Daily 

5-10 minutes 

Daily 

20 minutes 

Optional 

Length Unspecified 

Frequency and Length of 
Repeated Practice 

Regularly 3 
10 minutes 

Daily 

5-10 minutes 

Daily 

10-15 minutes 

Optional 

Length Unspecified 

Instructional Approach 

Teacher’s Role 

Facilitate student 
production of ideas 

Explain, model, facilitate 
production of ideas 

Explain, 

demonstrate, guide 

Explain, 

demonstrate, guide 

Classroom Interactions 

Teacher-Student 

(Student-Student) 

Teacher-Student 

(Student-Student) 

Teacher-Student 

Student-Text 

Pathway for Learning 

Between students and 
teacher 

From teacher to students, 
and 

between students 

From teacher to 
students 

From text to 
students 

Supports for Teachers 

How the Text Guides 
Teachers 

BLEND: Descriptive 
scripts guide teacher 
actions, with selective 
explicit scripts containing 
exact words to use 

BLEND: Detailed 
descriptive scripts and 
explicit guidance of 
teacher actions (rarely 
scripts teacher’s words) 

EXPLICIT: Fully 
scripted lesson; 
detailed description 
of teacher actions 
and room 
arrangements 

DESCRIPTIVE: 
Minimal description 
of teacher actions 


Note: The technical appendix further details the curriculum differences; DM=doing mathematics, PWC=procedures with 


connections, PWOC=procedures without connections, and M=memorization. DM and PWC are high-demand tasks; 
PWOC and M are low-demand tasks. 

Repeated practice in Investigations occurs at regular intervals at least once a week, but does not necessarily occur daily. 


Instructional approach refers to the teacher’s role during instruction, the types of activities 
in which students are expected to engage, and the nature of the classroom interactions. 
Investigations, Math Expressions, and Saxon have students and teachers interact with one 
another as they work on activities and concepts (Table 1). Investigations and Math Expressions 
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also emphasize student-to-student interactions and provide opportunities for students to work 
together and communicate their mathematics knowledge. With SFAW/enVision, in contrast, the 
predominant classroom interaction is between the student and the worksheet, workbook, or other 
curriculum materials. 

Supports for teachers concern the types of implementation guidance provided to teachers, 
including how each curriculum provides guidance and the topics of the guidance. The curricula 
provide guidance through either explicit scripts, descriptive scripts, or both. Explicit scripts 
specify what teachers should say. Descriptive scripts guide teachers’ and students’ actions or 
dialogue by describing what should be said, written, visually demonstrated, or done; or by 
providing both explicit and descriptive scripts. The topics of the guidance can include directing 
teachers’ actions, helping teachers understand student thinking, providing subject- specific 
content support, and clarifying curriculum designers’ rationale or intent. 

Investigations blends descriptions of teacher actions with selective explicit scripts of 
questions the teacher should ask or ways to respond to students (Table 1). Math Expressions is 
even more detailed in its descriptive script but less frequently specifies what the teacher should 
say. Unlike the other three, Saxon provides a fully scripted lesson containing almost everything 
the teacher should say, along with a detailed descriptive script. SFAW/enVision’ s guidance is 
more minimal and general than that of the other curricula. 

In terms of topics, both Investigations and Math Expressions provide guidance on a variety 
of teaching components, including mathematical concepts, student thinking, and ways to adapt a 
lesson for specific students. SFAW/enVision provides guidance on few topics, as does Saxon, 
which primarily focuses on classroom organization and management. 

Assigning the Curricula to Schools 

After selecting the curricula, we recruited schools to participate in the study. A total of 1 1 1 
schools from twelve districts enrolled in the study and agreed to participate for at least one 
year. 11 Of the 111 schools, 58 agreed to participate for a second year; the analyses presented in 
this brief are based on the 58 schools (from seven districts) that participated for two years. 

Though not a representative sample of all elementary schools in the United States, the 58 
schools included in these analyses are dispersed geographically and in areas with various levels 
of urbanicity. The participating schools also serve a higher percentage of students eligible for 
free or reduced-price meals than the average U.S. elementary school. 

At the outset of the study, we randomly assigned one curriculum to each school. The random 
assignment ensures that the four curriculum groups are similar in other ways and, therefore, any 
differences in classroom practices and student achievement can be attributed to differences in the 
effect of the curricula. The random assignment was conducted separately in each district, so each 
district implemented all four curricula. After random assignment, we introduced the school staff 
to the publishers of their assigned curriculum. Publishers then worked with the schools to deliver 
curriculum materials before the school year began and to schedule training for teachers. 
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How Teachers Implemented Their School’s Assigned Curriculum 

The differences between the curricula described above indicate how they differ when 
implemented as intended. But teachers and publishers may not always implement the curricula 
perfectly, which could influence the effect of the curricula on student achievement. To help shed 
light on this important issue, we analyzed implementation data collected by the study team 
through teacher surveys and classroom observations. 

The survey data indicated that nearly all teachers (at least 96 percent) in each curriculum 
group used their assigned program as their core curriculum. In addition, at least 70 percent of the 
teachers implemented a majority of the curriculum features as intended. According to the 
survey data, the amount of time spent on math instruction did not differ across the curriculum 
groups among lst-grade teachers. In 2nd grade, however, Saxon teachers spent one more hour 
per week on math instruction than did teachers using the other three curricula (6.1 hours versus 
5.1 hours). The appendix presents more information about implementation based on the survey 
data. 


Our classroom observation data show how well teachers implemented three aspects of the 
instructional approaches underlying the curricula, including the way teachers question students, 
respond to student answers, and guide practice. For each of these aspects, our observation data 
contain two to three related measures. 

For these aspects of the instructional approach, we found that teachers did what was 
expected for their assigned curriculum, with only two unexpected findings, both of which 
involve Math Expression, Saxon, and SFAW/enVision. First, based on our review of the 
curriculum materials, we expected Math Expressions teachers to ask fewer closed-ended 
questions and more open-ended questions than Saxon and SFAW/enVision teachers, but the 
classroom observation data do not show these patterns. Instead, as shown in Table 2, all three 
groups (Math Expressions, Saxon, and SFAW/enVision) asked similar numbers of closed-ended 
questions, and Math Expressions teachers asked as many or fewer open-ended questions 
compared with Saxon and SFAW/enVision teachers. Second, we expected Math Expressions 
teachers to be less likely than Saxon and SFAW/enVision teachers to immediately indicate 
whether a student’s answer was correct; instead, all three groups were equally likely to respond 
in this manner. 14 

Curriculum Effects 

To measure the effects of the curricula, we administered the math assessment from the Early 
Childhood Longitudinal Study-Kindergarten (ECLS-K) Class of 1998-1999 study. The ECLS-K 
assessment is an individually administered, adaptive, and nationally normed test that measures 
student achievement both within and across grades and meets accepted standards of validity and 
reliability (Rock and Pollack 2002). 15 The assessment includes both open-ended and multiple- 
choice questions designed to measure conceptual understanding, procedural knowledge, and 
problem solving in five math content areas: (1) number sense, properties, and operations; (2) 
measurement; (3) geometry and spatial sense; (4) data analysis, statistics, and probability; and 
(5) patterns, algebra, and functions. 16 
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Table 2. Implementation of Three Aspects of the Curricula in First- and Second-Grade Study Classrooms 17 

(average number of times each behavior occurred during one day of math instruction) 


Observation Item 

Investigations 

Math 

Expressions 

Saxon 

SFAW/ 

envision 

1. Teacher Questions 

Teacher asks closed-ended questions for which 
only one answer is acceptable 

18.2 

20.1 

21.0 

20.4 

Teacher poses open-ended questions that have 
more than one correct answer 

9.6 

4.3 

7.6 

4.8 

Teacher probes for reasoning or justification in 
response to student work/answer 

6.8 

4.9 

3.5 

4.7 

2. Teacher Responses to Student Answers 

Teacher repeats student answer in a neutral 
way without indicating correctness 

4.2 

0.3 

1.2 

1.6 

Teacher states whether student answer is 
correct without elaborating 

16.9 

19.8 

20.2 

18.7 

3. Types of Practice 

Number of problems on which the teacher 
guided practice 

5.2 

6.8 

10.8 

11.6 

Number of problems focused on review of 
previously learned material 

2.6 

6.0 

15.3 

4.4 


Note: The classroom observations were conducted by the study team using a protocol designed to distinguish differences 


between the curricula. The observation included an entire day of math instruction, including the math lesson and any 
morning meeting or calendar time, and the behaviors in the table had a possible range of 0 to 21 . The appendix 
further details the observation effort. All measures differ significantly across the curriculum groups at the 5 percent 
level of confidence. Statistics are based on 334 classroom observations conducted in 1st- and 2nd-grade 
classrooms. 


Two years of a curriculum. Among students who were taught using their school’s assigned 

curriculum for two years (in the 1st and 2nd grades), Math Expressions, Saxon, and 

SFAW/enVision improved math achievement by similar amounts. All three improved math 

18 

achievement more than Investigations by statistically significant amounts (Figure 1). 
Specifically, after being taught with their school’s assigned curriculum in 1st and 2nd grade, 
Math Expressions, Saxon, and SFAW/enVision students had an average spring 2nd-grade ECFS- 
K math score of 69.4, compared with 65.5 for Investigations students. Dividing this nearly 4- 
point difference by the 1 8-point standard deviation of the second-grade score indicates that Math 
Expressions, Saxon, and SFAW/enVision students scored an average of 0.22 standard deviations 
higher on the ECLS-K math test than Investigations students. This 0.22 difference (also known 
as an “effect size”) means that a study student at the 50th percentile in math would score 9 
percentile points higher as a result of being taught in 1st and 2nd grade with Math Expressions, 
Saxon, or SFAW/enVision instead of with Investigations. 19 

One year of a curriculum. Using the same set of districts and schools that participated in 
the study for two years, we examined the one-year effects students experienced by the end of 1st 
grade (Figure 1). The comparison of one- and two-year effects helps us understand whether the 

different teaching and learning approaches underlying the four curricula have implications for 

20 

students’ learning over time." 
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Figure 1. Math Expressions, Saxon, and SFAW/enVision improved 1st-through-2nd-grade math 
achievement by similar amounts, and all three outperformed Investigations 

1 st Grade ■ 2nd Grade 


Investigations 
Math Expressions 
Saxon 
SFAW/enVision 



Average Spring ECLS-K Math Score 


Note: The students included in this analysis entered the study at the beginning of 1st grade, at which point their school was 

randomly assigned to use one of the study’s four curricula. The starting point of 35 for the horizontal axis represents 
average math achievement of students at the beginning of 1st grade — that is, when they entered the study. The 
standard deviations of the spring 1st and 2nd grade scores equal 17 and 18, respectively. 


After one year (by the end of 1st grade), the achievement differences between 

Investigations, Math Expressions, and Saxon students were similar to the differences after two 

years (by the end of 2nd grade). In contrast, after one year, SFAW students scored 2.2 points 

lower than Math Expressions students on the ECLS-K test — a difference that is statistically 

significant. Dividing this 2.2 point difference by the 17-point standard deviation of the lst-grade 

score indicates that SFAW students scored an average of 0.13 standard deviations, or 5 percentile 

points lower than Math Expressions students. But, after the second year, achievement of SFAW 

and Math Expressions students was nearly equal. The effectiveness of SFAW/enVision therefore 

21 

increased as study students gained experience with this curriculum. " 

Switching curricula. Among the 111 schools that agreed to participate in the study for at 
least one year, 40 schools (Group 1) implemented their assigned curriculum in only the 1st grade 
during their first study year; the other 7 1 schools (Group 2) implemented their curriculum in both 
the 1st and 2nd grades, except for one school, which implemented it in only the 2nd grade. Our 
evidence about the effects of switching curricula is based on second graders in 37 (of the 71) 
Group 2 schools, for the following reason. Among the Group 2 schools, 37 reported using either 
Saxon or SFAW before the study. As such, the year before the study, the first graders in those 
schools were taught with either Saxon or SFAW. The next year, when these schools entered the 
study, they were randomly assigned to either continue using their pre-study curriculum (Saxon or 
SFAW), or switch to one of the other study curricula. As a result, the second graders in those 
schools either continued to be taught with one of the study’s curricula (Saxon or SFAW) or were 
taught with a different curriculum in the 2nd grade. We compared 2nd-grade achievement of 
students who did and did not switch from Saxon or SFAW to another study curriculum after 1st 
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grade, to understand how switching curricula affects students during the first year a new 
curriculum is used. The appendix provides more details about these analyses. 22 

This evidence suggests that switching between some of the study’s curricula does not harm 
student achievement and can even be beneficial. Students in the 7 schools who switched from 
SFAW to Saxon after 1st grade had an average 2nd- grade math score of 73.6, compared with 
68.4 for those (in 6 schools) who stayed with SFAW — a difference that is statistically significant 
(Figure 2). This effect is equivalent to moving a study student from the 50th to the 64th 
percentile in math achievement. The other types of switching we examined had no effect on 
student math achievement. Students in the 12 schools who switched from SFAW to 
Investigations or Math Expressions after 1st grade had similar 2nd-grade math scores: 68.4 for 
those who stayed in SFAW and 69.4 for those who switched to Investigations or Math 
Expressions (Figure 2). Similarly, students using Saxon for an additional year rather than 
switching to another curriculum had similar 2nd-grade math scores: 70.9 for students in 3 
schools who stayed with Saxon versus 70.4 for students in 9 schools who switched to another 

23 

curriculum (Figure 3).~ 


Figure 2. Average 2nd-Grade Math Achievement: Students Who Stayed with SFAW Through 2nd 
Grade and Those Who Switched to Another Study Curriculum After 1st Grade 


SFAW in 1st and 2nd Grades 


SFAW in 1st Grade; Investigations in 2nd 
SFAW in 1st Grade; Math Expressions in 2nd 
SFAW in 1st Grade; Saxon in 2nd 



55 60 65 70 75 


Average Spring 2nd-Grade ECLS-K Math Score 


Note: The students included in this analysis entered the study at the beginning of 2nd grade, at which point their school 

was randomly assigned to either continue using SFAW (their pre-study curriculum) or switch to another study 
curriculum. The starting point of 55 for the horizontal axis represents average math achievement of students at the 
beginning of 2nd grade — that is, when they entered the study. The standard deviation of the spring 2nd-grade score 
equals 15. The difference in average math achievement of students who stayed with SFAW through 2nd grade 
versus those who switched to Saxon after 1st grade is statistically significant at the 5 percent level; all other 
differences between students who stayed with SFAW versus those who switched are not significant. 
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Figure 3. Average 2nd-Grade Math Achievement: Students Who Stayed with Saxon Through 2nd 
Grade and Those Who Switched to Another Study Curriculum After 1st Grade 


Saxon in 1st and 2nd Grades 


Saxon in 1st Grade; Investigations in 2nd 


Saxon in 1st Grade; Math Expressions in 2nd 


Saxon in 1st Grade; SFAW in 2nd 



1 1 r 


55 60 65 70 


Average Spring 2nd-Grade ECLS-K Math Score 


75 


Note: The students included in this analysis entered the study at the beginning of 2nd grade, at which point their school 

was randomly assigned to either continue using Saxon (their pre-study curriculum) or switch to another study 
curriculum. The starting point of 55 for the horizontal axis represents average math achievement of students at the 
beginning of 2nd grade — that is, when they entered the study. The standard deviation of the spring 2nd-grade score 
equals 1 5. There were no statistically significant differences (at the 5 percent level) between the four groups. 


Discussion and Looking Ahead 

Many policymakers and education experts agree on the need for education reform and are 
focusing much of their attention on approaches to improving school governance, staffing, and 
content standards. Some research suggests that these approaches may have promise for 
increasing student achievement, but other studies show that an entirely different approach — 
changing a school’s curriculum — could lead to larger increases in student achievement at a much 
lower cost (Whitehurst 2009). 

In our study, one of the first to rigorously examine curriculum effects over multiple years, 
we compared the two-year effects of four math curricula that are widely used in the early 
elementary grades and differ in their approaches to teaching mathematics. Our results indicate 
that Math Expressions, Saxon, and SFAW/enVision improved lst-through-2nd-grade math 
achievement by similar amounts, and all three outperformed Investigations. 

Although these results help confirm that curricula can be an effective policy lever for 
improving student achievement in math, educators need at least three more pieces of information 
to help them use this lever successfully. First, researchers should look at the impacts of curricula 
on other assessments of student achievement that, like the one administered in this study, were 
designed to measure knowledge and skills that mathematicians and math educators feel are 
important for students to develop in early elementary school.' Given that about 44 states intend 
to implement new math assessments in response to the adoption of the Common Core State 
Standards, it is important to know how the curricula affect these new accountability assessments. 
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Second, researchers should examine curriculum effects beyond two years, along with the 
effects on students in other grades and subjects. The findings from this research may also be 
useful for certain states that limit the use of state funds for purchasing curricula. More 
comprehensive evidence about curriculum effects could be useful for these states’ decisions 
about which curricula to support. 

Third, educators need to know which features of curricula drive the effects we observed in 
our study. As mentioned in our descriptions of the study’s curricula, each program has key 
differences. For example, Investigations has higher demand tasks than Saxon, whereas the Saxon 
scripts are easier to use. Given that Saxon improved student achievement more than 
Investigations did, it is possible that Saxon balances task demand with ease-of-use more 
effectively than Investigations does. Of course, these features are only two important aspects of 
the curricula; other features also may play a role in curriculum effects. Knowing these features 
can help teachers select the right curriculum for their classrooms and implement it successfully. 
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ENDNOTES 


1 Previous study reports examined how the curricula affected students’ achievement during 
one year (Agodini et al. 2009, 2010). 

“ The publisher of SFAW revised and renamed the SFAW curriculum during the study. The 
new name of the curriculum is enVision. This change did not affect three of the seven districts 
that participated in the study for two years because their two years of participation preceded this 
change; however, it did affect the other four districts because it occurred halfway through their 
two years of study participation. We examined the implications of the SFAW-enVision change 
by comparing results based on students from all seven districts with those from the three districts 
that were not affected by the change. The pattern of results is similar across the two sets of 
districts; therefore, we report results based on the larger sample of seven districts. More details 
about these analyses are provided below and in the appendix. 

According to a 2008 survey (Resnick et al. 2010), Investigations, Saxon, and SFAW make 
up 32 percent of the curricula used by K-2 educators. Math Expressions is a newer curriculum 
and, therefore, its market share was not available in the 2008 survey. 

4 When a district/school adopts a new curriculum, it is typically implemented in each 
elementary grade at the same time. These whole-school curriculum changes every several years 
mean that a large fraction of students experience two math curricula during their elementary 
years. For example, if a K-5 school changes its math curriculum every six years, only students 
who are kindergarteners during the year of a curriculum adoption experience the same 
curriculum throughout their elementary years; students in grades 1 through 5 experience two 
curricula. Specifically, in these schools, about 17 percent of students experience the same 
curriculum throughout their elementary years; the other 83 percent experience two curricula. 

5 The following websites (accessed on May 13, 2013) provide information about 
California’s, Florida’s, and Texas’s math curriculum adoption processes, respectively: 
http://www.cde.ca.gov/ci/ma/im/, 

http://ritter.tea.state.tx.us/textbooks/adoptprocess/adoptioncycle.pdf, 

http://www.fldoe.org/bii/instruct_mat/pdf/cycle.pdf. 

6 Nine submissions were received; curricula submitted for consideration but not selected are 
not disclosed because proposals were confidential. 

1 

According to the National Research Council (2001), “Opportunity to learn is widely 
considered the single most important predictor of student achievement.” The goal of our analysis 
was to characterize the potential opportunities to learn mathematics available in each curriculum 
program. Traditionally, opportunity to learn (OTL) referred to the specific topics covered in the 
classroom and the time devoted to each (Floden 2002). Hiebert (2003) argued that you must also 
consider the nature and quality of that time, as well as the supports and structures that frame 
engagement with given tasks. Drawing on these perspectives, we identified three analytical 
categories to focus our analysis: (1) mathematical emphasis, (2) instructional approach, and (3) 
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supports for teachers. Our analysis does not include other aspects of instruction that research has 
found are important (such as formative assessments) because they were outside the scope of this 
study. 


o 

Stein, Grover, and Henningsen (1996) developed a scheme for categorizing tasks based on 
the extent to which they emphasize connections among concepts, procedures, and facts and their 
potential to engage learners with mathematical ideas. There are two types of high-demand 
tasks — “doing mathematics” and “procedures with connections” — and two types of low-demand 
tasks — “procedures without connections” and “memorization.” The appendix contains 
information about their prevalence in the curricula based on a sample of lessons from each 
program. 

9 Procedural fluency is one of the five components of mathematical proficiency proposed by 
the NRC (2001). It is defined as skill in carrying out procedures flexibly, accurately, efficiently, 
and appropriately (pp. 116). 

10 Hiebert and Grouws (2007) elaborate on the term “struggle with mathematics” by stating 
that it is “the opposite of simply being presented information to be memorized or being asked 
only to practice what has been demonstrated” (pp. 387-388). 

11 The 111 schools enrolled in the study beginning in either the 2006-07 or 2007-08 school 

year. 


12 

There is some risk that teachers overstated their implementation because they knew they 
were supposed to implement the curricula, but there is no reason to believe that the magnitude of 
inflation would be notably different among the four curricula. 

1 T 

The differences in math instructional time in 2nd grade are consistent with the publisher 
recommendations for instructional time. 

14 Although these results provide evidence that teachers used the curriculum assigned to 
their schools and implemented some aspects of the instructional approach as expected for their 
assigned curriculum, not all aspects of the instructional approach are measured with our data. For 
example, our observation data do not contain measures of how the teacher attends to student 
thinking and understanding. In addition, the data do not contain measures of the mathematical 
emphasis or the way in which the teachers utilized the supports available in their assigned 
curriculum. 

15 During the test selection process, members of the panel who reviewed the curriculum 
submissions also reviewed the tests under consideration to assess whether the tests provided an 
unfair advantage to one or more curricula, because we did not want to use an assessment that was 
biased toward some curricula; however, the review panel did not assess the tests for their 
alignment with the instructional objectives of the math curricula. The ECLS-K was considered to 
be the best option among the alternatives. 
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16 On the lst-grade test, about three-quarters of the items can be classified as number sense, 
properties, and operations; the remaining items are predominantly related to data analysis; 
statistics and probability; and patterns, algebra, and functions. On the 2nd-grade test, about half 
of the test is composed of items pertaining to number sense, properties, and operations; the other 
half is predominantly related to measurement; geometry and spatial sense; and patterns, algebra, 
and functions. Specific items included on the assessment are not provided because it is 
copyrighted. Rock and Pollack (2002) provide more details about the assessment, and the 
appendix of this brief provides more details about the process used to score the tests and 
information about test reliabilities. 

17 

The items presented in the table are a subset of the items on the observation protocol. The 
items selected for the table were considered by the study team to be those that most closely align 
with the aspects of the curricula examined through the comparative curriculum analysis. 

18 

As mentioned earlier, the participating schools serve a higher percentage of students 
eligible for free or reduced-price meals and a higher percentage of non-Hispanic black students 
than the average U.S. elementary school. More details about the study schools are provided in 
Table A. 8 of the appendix. 

19 The results for the SFAW/enVision curriculum group are difficult to interpret because 
some students in that group were taught with SFAW in both 1st and 2nd grades, whereas others 
were taught with SFAW in 1st grade and enVision in 2nd. To better understand the two-year 
effects of SFAW, we calculated results for the three districts that were not affected by the 
SFAW-enVision change, as mentioned above. In these districts, students in schools assigned to 
SFAW experienced that curriculum in both 1st and 2nd grades. These subgroup results are 
similar to the results based on the full sample — that is, after two years, Math Expressions, Saxon, 
and SFAW improved math achievement by similar amounts, and all three improved math 
achievement more than Investigations. Because the subgroup and full-sample results are similar, 
we report results based on the larger sample of seven districts, as mentioned above. The 
appendix provides more details about these results, along with a description of the enVision 
curriculum. 

Previous study reports examined one-year effects during 1st grade (Agodini et al. 2009, 
2010), but those results included students from other districts we did not include in this report 
because they did not participate in the study for two years. Therefore, to provide an accurate 
understanding of how the two-year effects in Figure 1 compare with the one-year effects in 
Figure 2, we calculated the one-year effects for the students who were actually taught their 
assigned curriculum for two years. 

“ Some teachers that taught the students in these analyses also gained experience 
implementing their school’s assigned curriculum as part of the study. Specifically, 43 percent of 
2nd-grade teachers in Group 2 schools were in their second year of study participation when 
teaching the study students during the 2nd grade. These Group 2 teachers represent 23 percent of 
all teachers (1st and 2nd grade) that taught the study students across the two years. 
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22 

“ First graders in some of the study schools also may have experienced a curriculum switch 
after kindergarten; however, we do not examine the effects of switching curricula between 
kindergarten and 1st grade because kindergarten math curricula are often less structured or 
defined. 

Because some curricula may be more effective in some grades than others, it would be 
useful for future research to examine the effects of switching curricula between other grades to 
assess whether our evidence is robust across grades. It also would be useful for future research to 
examine the switching-staying issue for Investigations and Math Expressions. Our study does not 
support these examinations because (1) only two schools were using Investigations before 
joining the study and both schools were assigned to continue using Investigations, and (2) no 
school was using Math Expressions before joining the study. 

' Recent evidence suggests that, under certain circumstances, the effects of mathematics 
curricula on student achievement at the middle school level are sensitive to the type of test 
examined. Tarr et al. 2008 examined curriculum effects on two tests: the Balanced Assessment in 
Mathematics (BAM) and TerraNova Survey (TNS). The test mattered only when measuring the 
effects of an NSF-funded curriculum when the curriculum was paired with moderate or high 
levels of a standards-based learning environment. The evidence is based on a quasi-experimental 
design, so it could be worthwhile to examine whether the outcome measure matters with a more 
rigorous approach. 
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For more information on the full study, please visit: 

http://ies.ed.gov/ncee/projects/evaluation/math curricula.asp 

To read the technical appendix, please visit: 

http://ies .ed. gov/ncee/pubs/20 1 340 1 9/pdf/20 1 340 1 9 app.pdf 
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